Many television programs include augmented reality graphics that serve to highlight an element that appears in the picture, present information to the viewer, or add a synthetic element to the picture. For example, a live sports video may be augmented with graphics highlighting a particular player action, or a virtual advertisement may be added to the walls of the venue at which the sports game is being played. In current systems, a video feed together with metadata is sent from a field venue to a central studio or other broadcast facility. The central location generates the augmented reality graphics using location and other information obtained from the metadata to position the graphics appropriately. The augmented reality graphics is then combined with the video into a single video stream with the augmented reality “baked in” and is broadcast to subscribers. In some situations, the augmentation is performed at the venue, for example in a broadcast truck, and then sent in broadcast-ready form to the central studio for distribution. The result is a one-size-fits-all program, in which all viewers see the same video, superposed graphics included. With the advent of smart set-top boxes and non-traditional viewing platforms such as smartphones and tablets, viewers are seeking more control over, and interaction with their devices.